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Motion estimator for reduced halos in MC up-conversion 

it 6. 02. 2001 



The invention relates to a method for detecting motion at a temporal 
intermediate position between previous and next images, in which a criterion function for 
candidate vectors is optimised, said function depending on data from both previous and next 
images and in which the optimising is carried out at the temporal intermediate position in 
5 non-covering and non-uncovering areas. 

The basic observation from which the current invention results is that an 
estimator estunating motion between two successive pictures from a video sequence, caimot 
perform well in areas where covering or uncovering occurs, as it is typical for these areas that 
the information only occurs in either of the two impages. Block matchers, as a consequence 
10 will always find large match errors even for the correct vector. 

The invention has the object to remove the above-cited problems, in general 
problems in calc\ilating vector fields from previous and/or next images. 

According to the invention this object is achieved in that the optimising is 
carried out at the temporal position of the next image in covering areas and at the temporal 
1 5 position of the previoxis image in xmcovering areas. 

Furthermore, the invention relates to an apparatus for detecting motion 
between previous and next images, comprising means for optimising a criterion function for 
candidate vectors, said function depending on data from both previous and next images and 
in which the optimising is carried out at the temporal intermediate position in non-covering 
20 and uncovering areas, wherein means for detection covering or uncovering areas are provided 
and that the optimising is carried out at the temporal position of the next image in covering 
areas and at the temporal position of the previous image in uncovering areas. Embodiments 
of the method and apparatus according to the invention are specified in sub-claims. 

The invention will be described in more detail v^th reference to the attached 



25 dravraigs. 



Figure 1 illustrates an example of a method for estimating motion. 
Figure 2 illustrates an ambiguous area due to covering; 
Figure 3 illustrates an ambiguous area due to uncovering; 
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Figure 5 illustrates an embodiment of a method for detectine 
foreground/background; 

Figure 6 illustrates another embodiment of a method for detecting 
foreground^ackground; 

Figure 7 shows a further embodiment of a method for detectmg 
fo^eground^ackground; 

Figu. 9 illustrates a. embotoem of a forcground/backg^uad de««or 
detector; "" """" "'"'^ f«-8«>undA«ckgro™d 

det^torand " '""^'^ '^^-^-'"''^-'—ae.gro.^d 

.5 incorporated. "'"^^-'^-^''^^ Reinvention is 

™''"'«'''°»'-'>-WHed in various technical fields. Examples are- picture 
«e converston. where an inrproved vector field according to the invention results in aire 

Pleastng.artei^fiee video strean..T^cal devices for thisappHcationareTVandrc 

0 arollingscene.'"*'^''"'"""'"''^"^™''--"'"^---^"' 

■"°''°°'^"<'«"=°'»P«=i«n where an improved vector field according to 
the ,nvent.on.es„.tsinlngher,„ali.,predictions and therefore inhighercon^io^J^ 
.mproved picture quaUty. An example is compression in MPEO. 

"here an improved vectorfleldaccordmg to the 
mvent-on results in a more feithfil extraction of objects and hence in an easier post, 
^^ng. Examples are: security cameta video analyses, video special effects and traffic 



background. 



picture edmcement in television techniques e.g. avoidance of blurring of 

scientific application, where an improved vector field according to the 

mventronresultsinabetterdataanalyses. Examples are satellitephotosofcloudsaad 
oceanography. 

It is recognised, that in the case of coverino oii • • . 

... . c v.dse oi covermg aU mformation in the current 

Ptctee ,s present in the previous picture, ,vhile in tire even, of .mcovering ^ n^c^s^ 
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while in an area of uncovering, the current picture contains all information of the previous 
one (locally around the uncovering area). Ergo, by modifying the match error calculation, 
controlled by a covering/uncovering detector, e.g. the one disclosed in WO 00/1 1863 from 
matching the current picture vnih the motion compensated previous picture (covering) to 

5 matching the previous picture with the motion compensated current picture (uncovering), any 
ambiguity in the estimator can be prevented. This is expected to yield more accurate and 
consistent vector fields and therefor reduced halo. This modification shall be elaborated a 
later part of the specification. 

As a side eflfect of this dynamically changing match calculation, the resulting 
1 0 unambiguous vector field is no longer valid for one moment in time, let alone the moment 
where the upconversion takes place, but this Validity moment' changes depending on the area 
imder consideration being a covering, imcovering or simple rigidly moving area. This effect 
can be eliminated with a second ingredient of this disclosure, described in a later part of the 
specification. 

1 5 Now the improved motion vector calculation using a fiillrsearch block- 

matching motion estimator algorithm to calculate the motion vectors, will be elucidated with 
reference to figure 1 . This algorithm is not a very cost-effective approach to the motion 
estimation problem, but the more effective one. The same principle is described in the article 
"True Motion Estimation with 3-D Recursive Search Block-Matching" of G. de Haan, P.W. 

20 A.C. Biezen, H. Huygen and O.A. Ojo, published m IEEE Tr. on Circuits and Systems for 
Video Technology, vol. 3, October 1993, pages 368-388 and the article "Sub-Pixel motion 
estimation with 3-D recursive search block-matching" of G, de Haan and P.W.A.C. Biezen 
published m Signal Processing: Image Communication 6 (1994) pages 220-239. 

In block-matching motion estimation algorithms, a displacement vector is 

25 assigned to the centre 



(X 



of a block of pixels B( X ) in the current picture n by searching for a similar block 



within a search area S A( X ), also centred at X , but in the previous picture « - 1 . The similar 
block has a centre, which is shifted with respect to X over the displacement vector D{X , «). 
30 To fmd b{X , n\ a number of candidate vectors C are evaluated by applying an error 

measure { C, X. n) to quantify block similarity. More formally, CS""^ is defined as the set of 
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candidates C , describing all possible (usually integer) displacements with respect to 



CS'°^={C|-N<Q<N,-M^Cy<M} (1) 

where i^andMare constants limitingSA(i-).AblockB(i-)centred at X andofsizeXby 
Y, consisting of pixel positions x = [ * in the present picture n, is defined: 



B(i-)={^|Xx-X/2 £x<Xx + X/2 Xy-Y/2<y ^ Xy +Y/2} (2 
The displacement vector DiX,n) resulting from the block-matching process, is a candidate 



D{x,n)e 

{C€C^\&(C,X,n) ^ eiV,X,n)^V€ CS^''} (3) 

If, which is the common case, the vector D{x,n) with the smallest matching 
error is assigned to all pixel positions x in the block B(X): 

Vx e BCX) : D(x,n) = D(x,n) (4) 

rather than to the centre pbcel only, a large reduction of the number of computations is 
achieved . - 

As an implication, consecutive blocks (B(X) are not overlapping. The error 
value for a given candidate vector C is a function of the luminance values of the pbcels in the 
current block and those of the shifted block from a previous picture, summers over the block 

(B( JT). A common choice, which we too shall use, is the sum of the absolute differences 
(SAD): 



X within the search area SA( X ) in the previous image: 




vector C which yields the minimum value of an error function ic,X,n 
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where a is a constant, 0 < a < 1, determining the temporal position between the two 
5 pictures, where the vector field has to be valid. 

Figure 2 shows more clearly what happens for all possible reference positions 
0 < a ^ 1 . Supposed that the upper line a corresponds to the position of the edge of the 
foregroimd object travelling through time. Then in the future picture we find the correct 
foregroxind or background velocities on either side of the velocity vector edge position Xg 
1 0 (position where the motion vector displace a significant spatial change), but the more we 
move the reference position a towards the passed picture, the larger the ambiguity region 
becomes, in which it is impossible to find the correct vector by imaging. Since all candidate 
vectors v^ll have a large SAD value, minimisation randomly selects a certain candidate 
which means that the resulting vector fields are almost impossible to correct. 
1 5 Similarly, from figure 3 we leam tiiat for uncovering regions an unambiguous 

motion estimation results, if we put tiae reference position a equal to 0, in other words we 
" project blocks firom the fiiture to the past. For rigid motion no errors occur if we put the 
reference position in the middle (for example a is 0,5), The decision whether an area is a 
covering or uncovering area could be done based upon the sign of the difference of the 
20 motion vectors' x-components taken fi-om either side of the vector edge Xe in the previously 
calculated vector field. 

The invention is based on the insight that in the case of e.g. covering, all pixel 
blocks present in and around the occlusion area in tiie fiiture picture can also be found in the 
past picture, however, in the past picture there are extra blocks, which no longer exist in the 
25 fiiture picture, because they become covered. A natural position to put the reference position 
a for block matching is at the temporal position of the future picture, since then for all blocks 
in and around the occlusion area a correct motion vector can in principle be foimd. 

The improvement proposed, results from the observation that in areas where 
covering occurs, correct motion vectors can only be estimated with an estimator using a = 1 . 
30 Similarly, for xmcovering only correct vectors are found with an estimator using a = 0. 

Vector estimators calculating vectors for intermediate temporal instances have 
difficulties witii botii covering and imcovering, be it that their worst case area of ambiguity is 
orvioiiof tv.a»> w fVw» wtTAmp f^ctim^toTQ (Thpi wnrfit case amhieiittv is least for the estimator 
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applying a = 0.5). The Figures 2 and 3 illustrate the effect of a on the size of the ambiguous 



area. 



The fu^t step is improving the block matcher for covering and uncovering 
situations, regardless the required value of a for the interpolated picture .results by changing 
5 equation 5 to: 

X ^ e B{x)\f{x - C, « - 1) - F{x, n)\ (6) 



10 in case an area of covering as indicated by the covering/uncovering detector, e.g. the one 
described in WO 00/1 1863 and to 



X ^ e B{x)\f{x, n-\)-F{x + C, n) [ (7) 
15 in the event of imcovering. 

In non-occlusion areas we take e.g. a = 0,5 and project the blocks 
bidirectionally from both the past and future pictures to the reference block position, i.e. 
compare them at the temporal position interpolated picture. For covering blocks we take a =1 
and match a block in the future picture in its ovm position with a block fetched with the 
20 candidate motion vector from the past picture. In uncovering areas of the picture we take a 
=0, in other words we matched the blocks in past picture with the blocks from the future 
picture. 

There are a number of ways in which occlusion or more specific covering and 
uncovering regions can be detected. 
25 The covering/uncovering detector may use a previous vector field, or a 

■previous calculation (iteration) of the current vector field, or use methods based on evaluation 
of match errors calculated between neighbouring pictures. 

A preferred covering/ uncovering detector looks at the signs of the velocities. 
If e.g. to neighbourmg velocities point towards each other we know that covering must occur. 
30 The other cases can be analysed accordingly and are summarised as follows 



vu - vd >5 (covering) 
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vu - vd < -5 (uncovering) 



in which vu is the velocity edge Xe, vd is the velocity below the edge and 5 is a small 
constant to avoid noisy detections. This method is extremely robust since it does not rely on 

5 the specific numerical values of the vectors. One will always have a more or less correct 
vector value on either side of the edge and even if the values are incorrect their difference 
still could be correct. Secondly a large band aroxmd the predicted edge is "guarded" since the 
estimations for covering respectively uncovering behave like the classical estimation 
(equation 7) in foreground FG regions i.e. outside the ambiguity triangle. This provides 

1 0 robustness against position deviations of the edge Xe, 

The benefit gained from switching a to 0 or 1, is that no incorrect vectors 
result from the minimisation, but the price to be paid is that the vectors obtained are not 
always in the correct block positions. E.g. the foreground FG edge (the block position where 
the vector changes from the backgroxmd BG to the foreground FG velocity) is foimd at the 

1 5 position Xe , in stead of at its real position U (see fig. 2, and 3). This is because of the time 
moment of the matching was incorrect (e.g. a = 0 in stead of e.g. a = 0,5). 

In fact the obtained vector field is not valid at any single time moment, but at 
three different time moments, depending on the position in the image, hence this method is 
called "tritemporal motion estimator". In general the vector field has to be retimed to the 

20 desired intermediate position (e.g. a = 0,5). In the specific case of a sequence in which the 
foreground FG object is stationary and the background BG moves behind it, no retiming of 
the tritemporal vector field is necessary, since the foreground FG edge does not move(XE = 
U). This happens often in film material where the cameraman tracks the main subject. 

As a consequence of the proposed modification, there are no ambiguities for 

25 the motion estimator, although we have sacrificed the correct temporal instance where the 
vector field is valid. This shall be detailed in the following sub-section. 

In a second step of the algorithm, the retimer, the time error is corrected. To 
this end, the vector field is 'projected* to the desired temporal instance, i.e. projected back in 
time for covering: 

30 D*({x, « - 1 + a) = i)(x -f (1 - a)D(x, n\ n) (8) 

and forward for uncovering: 

D « - 1 + a) = Z>(3c - o£>(x, n-\\n-\) (9) 
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This projection reintroduces ambiguous areas, i.e. areas to which no vector 
assigned. The origin of this ambiguity is that with one estimate it is unknown whether the 
discontinuity has moved along the line V. or along the line V in Figure 2, hence it is not 
trivial that the vector taken from e.g. position x above the discontinuity is the background 
vector which should be put in this position in the upconversion vector field. With help of the 
previous vector field however, it is possible to judge with which of the two possible vectors 
the edge moves, and hence which is the background and vdiich the foreground velocity. 

The retimer performs the following actions, it determines where the retiming 
should occur. It looks for a velocity edge Xe and marks a sufficiently broad occlusion region 
around this edge. Second it calculates how much blocks exactly should be corrected, by 
rounding the foreground FG velocity to block precision Third it determines which retimer 
conection action should be applied. 

Furthennore the retimer determines the starting position of the ambiguity 
area in the vector field at time n-1 +a by projecting the foreground velocity from the vector 
edge X, in the unambiguous vector field. A small refinement results if the position x^ of the 
edge in the intermediate vector field is not a result of shifting the estimates fix,m one vector 
field, but is calculated as the weighted (with a) average of the positions in the current and 
previous vector field. The retimer then fills the space between x, and x, with either the 
foreground or background velocity, depending on the side of the foreground region (left or 
right of X,) and the sign of the foreground velocity. For this strategy to work, a robust 
foreground/background detenmnation strategy is needed. 

It turns out that there are 8 different retimer cases depending on : 

1. covering versus uncovering 

2. the sign ofthe foreground FG velocity 

3. on which side of the velocity edge Xe the foreground FG is. This is shown in figure 4 in 
which as an example a = 0.5 for the intermediate image. 

The first upper part of figure 4 relates to the situation of covering, foreground 
FG at the bottom of the edge and foreground FG velocity Vfo< 0. In this situation two 
blocks foreground FG should be replaced by backgrounds BG blocks (fill BG). The second 
upper part of figure 4 relates to covering, foreground FG velocity Vpo > 0 and foreground FG 
at the bottom side of the edge. Only one background BG block should be replaced by a 
foreground FG block (fill foreground FG). The third upper part of figure 4 relates to 
uncovering, foreground velocity Vfg < 0 and foreground FG at the bottom side ofthe edge. 
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In this case only one background BG block should be replaced by one foreground FG block 
(fill foreground FG). The fourth upper part of figure 4 relates to uncovering, foreground 
velocity Vfg > 0 and foreground FG at the bottom side of the edge. In ttiis case only one 
foreground FG block should be replaced by a background BG block (fill background BG). 
5 The first lower part of figure 4 relates to covering, foreground FG velocity Vfg < 0 and 
background BG at the bottom side of the edge. Only one background BG block should be 
replaced by a foreground FG block (fill foreground FG ). The second lower part of figure 4 
relates to covering, foreground velocity Vpo > 0 and background BG at the bottom side of the 
edge. Only one foreground FG block should be replaced by a background BG block (fill 

10 background BG). The third lower part of figure 4 relates to uncovering, foreground velocity 
Vfg < 0 and foreground FG at the upper side of the edge. In this case one foreground FG 
block should be replaced by background BG block (fill background BG). The fourth lower 
part of figure 4 relates to uncovering, foreground FG at the top side of the edge and 
foreground velocity Vfg > 0. In this case one background BG block should be replaced by a 

1 5 foreground FG block (fill foreground FG ). Please note that in the example of figure 4 the 
intermediate image is at n + 1/2, but it is clear that the intermediate image could be at a 
distance a firom the previous image in which a could be varied. 

The retimer needs to know which velocity around an object edge is foregroxind 
and which is background. With the aid of a previous motion vector field it coxild be 

20 detera^ned which object regions or velocities belong to the foreground and which to the 
backgromd. A backgroxmd block is a block for which both the grey value pixels and the 
associated velocity vector disappear xmder covering. The correspondence of the vectors in 
foreground/background detectors are used, since they are judged to yield simpler, more 
reliable measures than pixel based measures . 

25 In a first strategy shown in figure 5, the average vector 

foreground/background determination, we make use of the fact that any vector v^^^ = 

kv ^-^ (I- where k is smaller than 1 and and v^^ are the velocities of the 

foreground and background objects at position 3c , fetches a background velocity from the 
previous vector field in the case of covering and a foreground velocity in the case of 
30 uncovering. The safest vector to use is the average vector v^^ = 0.5 v^^ + 0.5 v^^ . More 

formally, for covering we calculate the two possible x„ and ic^ of the edge in the previous 

image pair, e.g. for covering and a vertical edge at position in vector field n: 



10 



x„ =Xe+D(x- 



0 
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(10) 



and fetch the vector present at an intermediate position m the previous vector field (covering) 
in the ambiguous area: 



(11) 



If we need to fill in the foreground vector in the ambiguous area of the 
interpolation vector field, we choose between and the one which is most different 
from b^{Xyn), 



A variant of this first strategy fetches the background vector firom the future 
for uncovering: 

b,(x,n) = ri^^,ri^ (12) 

with: 



K =5] -D{x- 



^l^ 



,n-\)x^ =x,-Z)(x + l ,n-l) 



(13) 



A second strategy (see figure 6), the twosided self speed 
foreground/background determination, uses the fact that for uncovering, positions projected 
to the past with the background velocity have a higher probability of crossing towards the 
foreground region than when they are projected with v^. This is interesting when small 
relative velocities v^^^ - or inaccurately estimated vector fields occur. Because we do not 
know a priori which velocity is the background velocity, we project the two positions on 
either side of the edge with its own velocity (see Figure 3). As we can see for the ideal 
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case, the lower velocity changes from the background velocity at n to the foreground velocity 
at AJ - 1 . The probability that a block in foreground in n projects to foreground in n- 1 is so 
high that for practical reasons we can consider it to be 1. All the other probable decisions are 
shown in table 1. 



Table 1 



Upper block is in reality: 


foreground 


backgroxmd 


Decision for upper block 


FG 


FGorBG 


Decision for lower block 


FGorBG 


FG 



In case the two projections yield the same (foreground) vector, we have a 
certain determination. If this vector equals the vector of the upper block, this vector is the 
foreground velocity vector and vice versa. In case the vectors are different, the method was 
unsuccessful, and yields an uncertain determination. A similar projection towards the future 
can be applied for the case of covering. 

A third strategy (see figure 7), the edge projection foreground/backgrotmd 
determination, checks for e.g. covering whether the edge between v^^ and v^^ in the 
previous image n is present at position a or b (see Figure 2). If the edge is detected at position 



a, v-^ =D(jc- 



ln±l) and vice versa. Care should be taken that the velocities in « - 1 are 



the same velocities as in w, since other velocity edges can occur in the vicinity of the 
projection. Obviously again the principle can be applied by substitutmg imcovering for 
covering and the future for the past. 

It should be noted that the strategies can be enhanced by incorporating the 
match errors. In case a crossing to the foreground region occurred, the match errors of the 
vector in that block should be low. In case we project to a background region that was 
erroneously allotted a foreground vector in the previous image, the errors should be higher. 

In figure 8 an apparatus for detecting motion at a temporal intermediate 
position between previous and next images according to an embodiment of the invention is 
shown. This apparatus comprises means 1 for optimising a criterion function for candidate 
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vectors, which function depends on data from both previous and next images. Such an 
optimising means could be implemented as described in the article "True Motion Estimation 
with 3-D Recursive Search Block-Matching" of G. de Haan, P.W. A.C. Biezen, H. Huygen 
and O.A. Ojo, published in IEEE Tr. on Circuits and Systems for Video Technology, vol 3, 
5 October 1993, pages 368-388, To the input of the means 1 for optimising a criterion function 
for candidate vectors an image signal J is applied, while at the output of means 1 a motion 
vector MV could be derived. 

Said image signal E is also supplied to means 2 for detecting 
covering/xmcovering areas, the output of which is connected to the means 1 for optimising a 

10 criterion function for candidate vectors. Said means 2 for detecting covering/uncovering 
areas is for example disclosed in WO 00/1863. 

The means 1 for optimising a criterion fimction for candidate vectors is 
provided with switching means controlled by means 2 for detecting covering/uncovering 
areas such, that the optimising by means 1 is carried out at the temporal intermediate 

1 5 position in non-covering and non-uncovering areas, whereas the optimising is carried out at 
the temporal position of the next image in covering areas and at the temporal position of the 
previous image in uncovering areas. 

The previous image is shifted over a fraction a times the candidate vector and 
the next image is shifted over 1 - a times the candidate vector and the fraction a may 

20 change within the image period. The above-mentioned criterion function is a match error 
which is minimised. Said match error is also a function of the fraction a . 

The means for optimising the match error is arranged such that the fraction a 
is controlled by a covering/uncovering detector in the matching process. Preferable the 
fraction a is set to 1 in case of covering and set to 0 in case of uncovering. 

25 The means 2 for detection covering/uncovering areas decides preferably on 

data in a previous image to the fraction a in the cxirrent estimation. 

The image signal J is also applied to the foreground/background detector 4 the 
output of which controls the retimer 3. The output of the retimer 3 is connected to the means 
1 for optimising the criterion function. 

30 The retimer 3 determines a velocity edge X e in the image signal and marks an 

occlusion area around said edge. The retimer 3 controls the means 1 for optimising such that 
in said occlusion area a foreground velocity is replaced by a backgroimd velocity or 
reversibly dependent on the occlusion is a covering or uncovering area, the sign of the 
foreground velocity and on which side of the velocity edge Xe foreground is. 
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In figure 9 an embodiment of a foreground^ackg^ound detector according to 
the invention is shown. This detector comprises calculating means 5 and 6 to which the 
image signal J is supplied for carrying out a calculation at a position jc, of a velocity edge. 
The calculating means 5 calculates a first position in the previous (covering) or next 
5 (uncovering) image by shifting Xj over the first vector at one side of the edge, while the 
calcxilating means 6 calculates a second position 3c^ in the previous (covering) or next 
(uncovering) image by shifting x^ over the second vector at the other side of the edge. The 
choice between previous and next images is carried out by a covering/uncovering detector 7, 
while it is clear that for this detector the means 2 for detecting covering and uncovering areas 
10 2 of figure 1 could be used. 

The outputs of the calculating means 5 and 6 are connected to the inputs of 
calculating means 8, which calculates a third intermediate position between x^ and 

The foregroxmd/backgroxmd detector of figure 9 is fiarther provided with 
1 5 fetching means 9, one input of which is connected to the output of means 8 for calculating the 
third position, while to the other input of which the image signal J is supplied. From the 
output of the fetching means 9 the background velocity Vbg could be derived. The fetching 
means 9 fetches with the velocity at the third position the background vector firom the 
previous (covering) or next (uncovering) image. This vector is filled in those regions of the 
20 image in the environment of the edge, to which no vector is projected, in case the background 



vector v^^ should be filled in and the vector chosen between D{x - In) and D(x + 



^1^ 



which is most different fi-om v^^ is filled in, in case a foreground vector v^^ should be filled 
in. According to a fiirther elaboration the third intermedidate position is (3c^ 4- )/2. 

Another embodiment of a foreground/background detector is shown in figiire 
25 10. This detector comprises projecting means 10, 11 to the input of which an image signal J 
is supplied. These projecting means 10, 1 1 are controlled by a covering/uncovering detector 
12 to which also the image signal J is supplied. Said detector 12 could be substituted by the 
means 2 for detecting covering and uncovering areas shown in figure 1 . The control 
operation of the covering/uncovering detection is such that the projecting means 1 0 projects 
30 one position at one side of the velocity edge to the previous (covering) or next (uncovering) 
image, while the projecting means 1 1 projects the position of the other side of the velocity 
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edge to the previous (covering) or next (uncovering) image. Identification means 14 is 
connected to the projecting means 10 and 1 1, at the output of which the background velocity 
and the foreground velocity F^^ could be derived. This identification means 14 identifies 
a background velocity as a velocity which crosses the velocity discontinuity and projects to a 
5 foreground velocity in the previous picture, whereas a foregroxmd velocity projects to itself. 

Preferably a checking means 13 is connected to a projecting means 10 and 11, 
which means checks if the two projections yield the same vector. If so, the identification is 
certain. 

In figure 1 1 a further embodiment of a foreground/background detector is 
10 shown. This detector is also controlled by a covering/uncovering detector 17, which also 

could be substituted by the means 2 for detection of covering and uncovering areas of figure 
1, In the detector of figure 1 1 an image signal J is suppUed to the input of a projecting means 
16, which under the control of the covering/uncovering detector projects the discontinuity to 
the previous vector filled in the covering situation and to the fixture vector field in the 
1 5 uncovering situation. Furthermore, means 1 8 for determining the velocity on one side of the 
discontinxxity and means 1 9 for determining the velocity on the other side of the discontinuity 
are provided. The determined velocities are supplied to testing means 20 from the ou^ut of 
means 18 and 19. Said testing means 20 is connected to the projecting means 16 for testing 
whether the edge has moved over tiie first vector on one side of the discontinuity or over the 
20 second vector on the other side of the discontinxiity. The testing means 20 delivers the 
background vector , which background vector is the second (first) vector in case the 

discontinuity moves with the first (second) vector. 

Furthermore filling means could be cormected to the testing means 20, which 
filling means fills the first (second) vector in those regions of the projected vector field in the 
25 environment of the discontinuity, to which no vector is projected, in case a foreground vector 
should be filled in and the other vector is filled in, in case a background vector should be 
filled. 

In figure 12 an image display apparatus is shown. This apparatus comprises an 
apparatus 21 for detection a motion vector at a temporal intermediate position between 
30 previous and next images. An image signal J is supplied to the apparatus 21 and also to 
means 22 for interpolating image parts between previous and next images. The detected 
motion vectors at the output of the apparatus 21 are supplied to the input of interpolating 
means 22. The output of the interpolating means 22 is applied to a cathode ray tube 23, The 
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apparatus for detecting a motion vector could be implemented according to one of the above 
described apparatus. 
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1 , Method for detecting motion at a temporal intermediate position between 
previous and next images, in which a criterion function for candidate vectors is optimised, 
said function depending on data from both previous and next images and in which the 
optimising is carried out at the temporal intermediate position in non-covering and non- 

5 uncovering areas, characterised in that the optimising is carried out at the temporal position 
of the next image in covering areas and at the temporal position of the previous image in 
uncovering areas. 

2. Method according to claim 1 , wherein the previous image is shifted over a 

10 fraction a times the candidate vector, the next image is shifted over 1 -a times the candidate 
vector and the fraction a may change within the unage period. 

3^ Method according to claim 1 , or 2, wherein the criterion function is a match 

error which is minimised. 

15 

4. Method according to claim 2 or 3, wherein the fraction a is controlled by a 

covering/xmcovering detector in the matching process. 

5^ Method according to claim 4, wherein the fraction a is set to 1 in case of 

20 covering and set to 0 in case of uncovering. 

6. Method according to claim 4 or 5, wherein the covering/uncovering detector 

decides on data in a previous image to the fraction a in the current estimation. 

25 7. Method according to one of the preceding claims, wherein a velocity edge Xe 

is determined, an occlusion area is marked around said edge and in said area foreground 
velocity is replaced by background velocity or reversibly dependent on the occlusion is a 
covering or uncovering area, the sign of the foregroxmd velocity and on which side of the 
velocity edge Xe the foreground is. 
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Method according to claim 7, wherein at the position ic, of a velocity edge 

• a&stposition the previous (covering) or next (uncovering) image is 
calculated by shifting x, over the first vector at one side of the edge 

• a second position x, in the previous (covering) or next (uncovering) image is 
calculated by shifting x, over the second vector at the other side of the edge 

• and a third intermediate position between x^ and x, is calculated 

• while finally, the vector fetched with v,, at the third position in the previous 
(covering) or next (uncovering) image is filled in those regions of the image in the 
environment of the edge, to which no vector is projected, in case the background 

vector V,, should be filled in and the vector chosen between 5(ic-^^j,„)and 

^(x+Q,«) which is most different from is filled in, mease a foreground 
vector v^^ should be filled in. 

Method according to claim 8. wherein the intermediate position is (ic„ + 



-asa 



Method accordtag to claim 7, wherein a backgram„I velocity is identified „ . 
velocy wUch cro^ fte velocity discontinuity and projects K, a foreground velocity i„ the 
previous picture, whereas a foreground velocity projects to itself. 

1 1. Method according to claim 7. wherein near edges it is tested whether the 

mentions! edge has moved over the fita vector on one side of the edge, or over die second 
vecuir on the other side of the edge, in case die edge moves with die first (second) vector die 
«cond (first) vector is filled in those regions of die projected vector field in die enviromnen, 
of the edge, to which no vector is projected, in case a background vector should be filled 
in, and the other vector is filled in, m case a foreground vector should be filled. 
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12. Method according to claim 10, wherein the crossing from a backgroxmd region 

to a foreground region in the previous image is verified by the match error of the vector in 
that block. 

5 13. Apparatus for detecting motion at a temporal intermediate position between 

previous and next images, comprising means (1) optimising a criterion function for candidate 
vectors, said function depending on data from both previous and next images in which the 
optimising is carried out at the temporal intermediate position in non-covering and non- 
uncovering areas, characterised in that means for detection covering or uncovering areas (2) 
10 are provided and that the optimising is carried out at the temporal position of the next image 
in covering areas and at the temporal position of the previous image in uncovering areas. 

14, Apparatus according to claim 13, wherein the previous image is shifted over a 
fraction a times the candidate vector, the next image is shifted over 1 -a times the candidate 

1 5 vector and the fraction a may change within the image period. 

15. Apparatus according to claim 13, or 14, wherein the criterion function is a 
match error which is minimised. 

20 16. Apparatus according to claim 14 or 15, wherein the fraction a is controlled by 

a covering/uncovering detector (2) in the matching process. 

17. Apparatus according to claim 1 6, wherein the fraction a is skt to 1 in case of 
covering and set to 0 in case of uncovering. 

25 

18. Apparatus according to claim 16 or 17, wherein the covering/xmcovering 
detector (2) decides on data in a previous image to the fraction a in the c\jrrent estimation. 

19. Apparatus according to one of the preceding claims, wherein a velocity edge 
30 Xe is determined, an occlusion area is marked around said edge and in said area foreground 

velocity is replaced by background velocity or reversibly dependent on the occlusion is a 
covering or uncovering area, the sign of the foreground velocity and on which side of the 
velocity edge Xe the foreground is. 
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20. Apparatus according to claim 19, wherein calculation means (5,6,8) are 

provided for, at the position of a velocity edge, calculating 

• a first position in the previous (covering) or next (uncovering) image by 
shifting over the furst vector at one side of the edge 

5 • a second position in the previous (covermg) or next (uncovering) image by 

shifting X, over the second vector at the other side of the edge 

• and a third intermediate position between x^ and , 

• while finally, the vector fetched with v^^ at the third position in the previous 
(covering) or next (uncovering) image (9) is filled in those regions of the image in 

10 the environment of the edge, to which no vector is projected, in case the 

background vector v^^ should be filled in and the vector chosen between 



D(x-. 

0 



n . . . ^ ri 



n) and D(x+\ ^ In) which is most different fi"om v^^ is filled in, in 
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case a foreground vector v^^ shoxxld be filled in. 

Apparatus according to claim 20, wherem the intermediate position is (x^ + 



22. Apparatus according to claim 1 9, wherein means (1 0, 1 l)are provided for 
projecting two positions on either side of the edge to the previous (covering) or next 

20 (uncovering) image, in which a background velocity is identified (14) as a velocity which 
crosses the velocity discontinuity and projects to a foreground velocity in the previous 
picture, whereas a foregroimd velocity projects to itself 

23. Apparatus according to claim 19, wherein means (20) are provided for testing 
25 near edges whether the mentioned edge has moved over the first vector on one side of the 

edge, or over the second vector on the other side of the edge, in case the edge moves with the 
first (second) vector, the second (furst) vector is filled in those regions of the projected vector 
field in the enviroimient of the edge, to which no vector is projected, in case a background 
vector v^^ should be filled in, and the other vector is filled in, in case a foreground vector 
30 should be filled. 
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24. Apparatus according to claim 22, wherein verification means are provided for 
verifying the crossing from a background region to a foreground region in the previous image 
by the match error of the vector in that block. 

25. Image display ^paratus comprising detection apparatus (2 1 ) for detection a 
motion vector according to claim 13, means (22) for interpolating image parts connected to 
said detection apparatus (21) and a display device (23) connected to the interpolating means. 
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An apparatus for detecting motion at a temporal intermediate position between 
a previous image and a next image has optimizing means (1 ) for optimizing a criterion 
function for candidate vectors, whereby the criterion fimction depends on data from both the 
previous and the next image. The motion is detected at the temporal intermediate position in 
non-covering and in non-uncovering areas. The apparatus according to the invention has 
means for detecting covering and uncovering areas (2) and has its optimizing means being 
arranged to carry out the optimizing at the temporal position of the next image in covering 
areas and at the temporal position of the previous image in uncovering areas. 
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